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DIAGNOSING  MUNITION  FAULTS: 

THE  EMMA  APPROACH 

Bany  E.  Mullins,  Capt,  USAF 
Air  Force  Armament  Laboratory  (AFATUFXG) 
Eglin  Air  Force  Base,  Florida  32542-5434 


ABSTRACT 

The  Expart  Missile  Maintenance  Aid 
(EMMA)  program  ia  diacuaaad  in  thia  papar. 
The  papar  addraaaaa  auch  iaauaa  aa  how  tha 
field-level  expert  ayatam  prototypaa  wara 
structured  and  avaluatad  aa  wall  aa  tha 
excel lant  raaulta  of  tha  avaluationa.  It 
waa  found  that  tha  uaa  of  axpart  system*  la 
vary  amenable  and  banaficial  to  diagnosing 
munition  faults  in  tha  field.  Currant  work 
on  tha  dapot-laval  prototypaa  ia  diacuaaad 
aa  well  aa  iaauaa  related  to  using  DO D- STD- 
21  67  to  document  tha  davalopaiant  of  axpart 
ayatana.  Whan  properly  tailored,  DOD- STD- 
21  6  7  can  be  an  effactlve  means  of 
documenting  and  monitoring  tha  work 
performed  in  developing  an  expert  system. 
Tha  papar  concludes  with  a  discussion  of 
future  plana  for  a  follow-on  program  and 
other  areas  of  concern. 

iirrwoppcTiom 

Weapon  systems  of  today  are  undoubt¬ 
edly  benefiting  from  technology  advances 
and  justifiably  so.  Munitions  era  becoming 
more  sophisticated,  autonomous,  and 
‘smarter"  an  a  result  of  this  tscbnology. 
Electronically  sophisticated  monitions  are 
quickly  infiltrating  the  Department  of 
Defense  arsenal  of  weapons-'  in  order  to . 
combat  the  ever-increasing  number  of 
sophisticated  weapon  systems  of  our 
adversaries.  Siaqple  bombs  are  becoming 
relics  of  yesteryear  with  technology 
pushing  our  weapons  towards  autonoay. 

However,  some  shortcomings  can  be 
associated  with  incorporating  new 
technology  into  current  and  future  weapon 
systems.  Munition  maintenance  will  suffer 
as  a  consequence  of  this  technology.  The 
technology  advances  that  have  improved  the 
effectiveness  of  munitions  are 
simultaneously  complicating  the  maintenance 
of  these  munitions  by  increasing  the 
functionality  of  the  munition  typically  at 
the  expense  of  the  munition's 
suiintainability.  Munition  test  equipment 
and  aasociated  test  software  do  not 
adequately  diagnose  faults.  Automatic  test 
equipment  (ATE)  is  plagued  by  high  false 
alarm  rates.  Guidance  and  control  sections 
returned  to  the  depot  are  currently 
experiencing  approximately  26  to  44  percent 
retest  OKs.  Many  faults,  as  many  as  one 
out  of  evesy  four  (i.e.,  25  percent), 
cannot  be  detected  by  ATE.  Sequential 
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testing  (i.e.,  sequentially  applying  a 
series  of  predetermined  tests  to  the  faulty 
munition  regardless  of  a  priori  knowledge 
of  the  munition)  is  typically  how  ATE 
performs  diagnostic  testing.  This  limits 
the  diagnostic  capabilities  of  the  test 
equipment.  Additionally,  current  munition 
ATE  cannot  diagnose  beyond  multiple  linked 
cosqwnents . 

Another  aspect  of  munition  maintenance 
that  must  be  addressed  when  dealing  with 
munition  reliability  ia  the  personnel 
shortage.  The  current  shortage  of  skilled 
munition  maintenance  technicians  is  a 
serious  problem.  Demographic  projections 
indicate  that  this  dilemma  will  not  subside 
in  the  near  future.  Since  experienced 
technicians  are  able  to  diagnose  a  fault 
quicker  and  more  reliably  than  a  novice, 
the  knowledge  .acquired  by  the  experienced 
technician  (expert)  throughout  the  years 
should  be  captured  so  that  this  knowledge 
can  be  used  by  novice  technicians  during 
future  diagnostic  sessions. 

Artificial  intelligence  (AI) 
technology  is  one  approach  to  increasing 
tha  reliability  and  maintainability  of 
existing  and  future  weapon  systems.  One 
popular  and  heavily  cited  definition  of 
artificial  intelligence  is  provided  by  Dr. 
Elaine  Rich,  University  of  Texas  at  Austin. 
She  defines  Al  as  followsi  ‘Artificial 
intelligence  is  the  study  of  how  computers 
do  things  at  which,  at  the  moment,  people 
are  better"  (Rich,  1983 il).  A  subset  of  Al 
is  a  field  called  expert  systems.  This 
area  of  AX  has  esMrged  recently  with  the 
greatest  amount  of  success  (Hayes-Roth  et 
al.,  1983sxi).  Donald  Waterman  defines 
expert  systems  as  ‘sophisticated  computer 
programs  that  manipulate  knowledge  to  solve 
problems  efficiently  and  effectively  in  a 
narrow  problem  area”  (Waterman,  1966txvii). 
Tactical  munition  maintenance  is  one  such 
area  where  a  narrow  problem  area  can  be 
defined  where  a  computer  could  potentially 
perform  better  than  a  human. 

EMMA’ 

EMMA  is  a  research  effort  sponsored  by 
the  Air  Force  Armament  Laboratory,  Air 
Force  Systems  Command  at  Eglin  Air  Force 
Base  (AFB),  Florida.  It  is  a  first  attempt 
to  enhance  maintenance  of  a  tactical  muni¬ 
tion  at  the  field  and  depot  level  by  using 
AI  techniques.  The  ultimate  goal  of  EMMA 
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is  to  assist  a  novice  munition  maintenance 
technician  isolate  and  diagnose  electronic, 
electromechanical,  and  mechanical  equipment 
faults  of  a  single  munition  to  the  board 
/chassis/component  level  more  quickly  and 
consistently  than  the  best  human  expert 
using  the  best  currently  available  ATE.  To 
this  and,  EMMA  augments  existing  ATE  with 
an  expert  system  that  captures  the  knowl¬ 
edge  of  design  and  aiaintenance  experts, 

EMMA  is  a  30-month  effort  split 
into  two  phases.  The  program  is  structured 
such  that  the  maintenance  concept  for 
munitions  is  followed.  Unlike  aircraft 
maintenance  with  three  levels  of 
maintenance,  munition  maintenance  only  has 
two  levels  —  field  and  depot.  Phase  1, 
September  1986  -  July  1987,  addressed  the 
field-level  maintenance  of  tactical 
munitions  and  ultimately  resulted  in  two 
field-level  expert  system  prototypes. 
Phase  2,  August  1987  -  April  1989,  focuses 
on  depot-level  maintenance  and  will  produce 
two  depot-level  expert  system  prototypes. 
Since  depot-level  diagnostic  activities  are 
more  indepth  and  detailed  than  the  field, 
this  phase  is  expected  to  be  more  difficult 
and  of  greater  complexity.  This  accounts 
for  the  greater  time  allotted  to  this 
phase.  The  prototypes  fram  both  phases  are 
targeted  for  use  by  the  maintenance 
technicians.  Since  EMMA  is  constrained  by 
schedule  and  money,  the  number  of  tests 
developed  under  this  effort  is  limited,  yet 
sufficient,  to  demonstrate  concept 
feasibility  of  using  expert  systems  for 
munition  maintenance. 

EMMA  draws  on  many  different  types  of 
knowledge  and  information  to  perform  the 
diagnosis  of  the  faulty  sninition  including 
maintenance  rules  or  Technical  Orders 
(TOs),  maintenance  technician  practices 
(heuristics).  Unit  Under  Test  (UUT)  design, 
existing  test  equipment  capabilities,  fail¬ 
ure  rates,  and  test  costs.  Figure  1  de¬ 
picts  how  this  knowledge  is  brought  to  bear 
on  the  problem  of  diagnosing  the  faulty 
munition.  First,  the  symptoms  are  derived 
from  the  test  equipment  and  technician 
observations.  This  information  is  supplied 
to  the  expert  system  via  a  sophisticated, 
user-friendly  interface.  The  expert  system 
then  employs  the  knowledge  stored  In  the 
knowledge  bases  and  derives  a  repair 
strategy  which  is  displayed  to  the 
technician  using  the  EMMA  computer  screen. 

EMMA  is  a  dual  contract  effort 
performed  by  Raytheon  Company,  Missile 
Systems  Division  in  Bedford  Massachusetts, 
and  Rockwell  International  Corporation, 
Autonetics  Sensors  and  Aircraft  Systems 
Division  in  Anaheim,  California.  Both 
contractors  were  required  to  develop  a 
field  and  depot  EMMA  prototype  resulting  in 
a  total  of  four  prototypes  (two  for  each 
contractor)  and  were  allowed  to  select 
their  candidate  vehicle  for  the  EMMA 
program  within  specified  limits.  Raytheon 
selected  the  AIM-7F  Sparrow  missile  as 
‘"•(r  candidate  munition.  Rockwell  chose 
the  GBU-15  modular  glide  bomb. 


Figure  1.  EMMA  Expert  System 


THE  AIM-7F  FIELD-LEVEL  EMMA  PROTOTYPE 

The  Raytheon  field-level  (phase  1) 
EMMA  prototype  was  designed  to  enhance  the 
field-level  maintenance  of  the  AIM-7F 
missile  by  augmenting  the  missile's  test 
set.  All  references  to  the  word  "EMMA"  in 
this  section  refer  to  the  Raytheon  AIM-7F 
version  of  EMMA.  The  field-level  test  set 
for  the  AIM-7P  is  the  AN/DSM-162  test  set. 
EMMA  is  hosted  on  a  Symbolics  3670  LISP 
machine  running  the  expert  system  shell  ART 
(Automated  Reasoning  Tool).  ART  provides  a 
production  language  that  is  primarily  rule- 
based.  Consequently,  EMMA  was  developed 
using  the  rule-based  approach.  The 
Symbolics  computer  is  connected  to  the 
AH/DSM— 162  test  set  via  an  RS-232  cable. 
Figure  2  illustrates  the  major  components 
of  the.  EMMA  system  and  how  they  are 
interconnected . 

The  RS-232  cable  allows  EMMA  to 
operate  in  three  modes  —  automatic,  semi¬ 
automatic,  and  manual.  The  distinguishing 
characteristics  of  these  inodes  is  the  level 
of  automation  EMMA  is  allowed  during  the 
diagnostic  session.  The  automatic  mode 
uses  the  RS-232  interface  to  allow  EMMA  to 
direct  the  diagnostic  testing  and 
resequencing  of  tests.  EMMA  automatically 
accepts  data  from  the  test  set  via  the  RS- 
232  cable,  performs  the  fault  isolation, 
and  directs  the  test  set  to  perform  addi¬ 
tional  tests  if  required  until  the  fault  is 
detected  or  all  tests  pass.  If  a  fault  is 
detected  during  automatic  operation,  the 
user  may  switch  to  semi-automatic  mode  for 
closer  control  over  the  testing  and  the 
ability  to  query  after  each  test  segment. 

The  semi-automatic  mode  operates 
similarly  to  the  automatic  mode  with  one 
exception.  This  mode  stops  execution  of 
EMMA  at  the  completion  of  each  unique  test 
segment.  This  allows  the  technician  to 
query  EMMA  recommendations  using  the 
explanation  capability.  Another  advantage 
of  the  automatic  modes  (semi  and  full)  is 
data  integrity.  Since  EMMA  passes  the  data 
between  the  test  set  and  the  Symbolics 
computer  via  tha  RS-232  cable,  the  data  are 
more  likely  to  remain  valid  aa  opposed  to 
transferring  data  via  a  technician  who 
could  inadvertently  introduce  errors. 
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Figure  2.  AIH-7F  Field-level 
EMMA  Prototype 


The  laet  node  ie  manual.  This  node  ia 
provided  in  caee  an  US-232  connection  ia 
not  poaaible.  Aa  the  name  inpliea,  all 
interactiona  between  the  teat  aet  and  the 
Synbolics  conputer  anat  be  performed  manu¬ 
ally  by  the  technician.  EMMA  will  direct 
the  technician  to  perform  the  appropriate 
actions  to  the  teat  aet  and  wait  for  the 
response.  The  technician  enters  the 
responses  from  the  test  set  into  EMMA. 

Aa  with  moat  expert  systems,  EMMA  ia 
able  to  explain  its  reasoning  process  to 
the  user  (technician)  by  explaining  its 
fault  detection  and  resequencing  logic.  In 
other  words,  EMMA  explains  a  detected  fault 
and  why  a  certain  test  is  being 
reconsmnded.  Two  levels  of  explanation  are 
available  depending  upon  the  experience  of 
the  technician.  The  technician  may  request 
an  explanation  during  any  phase  of  the 
diagnostic  process.  This  allows  the 
technician  to  query  EMMA  during  a 
consultation  which  heightens  the 
technician's  understanding  of  what  EMMA  is 
doing  while  simultaneously  providing  the 
technician  with  a  valuable  training  aid. 

One  of  the  moat  critical  aspects  of 
any  software  system  is  its  user-friendli¬ 
ness.  If  the  system  is  difficult  to  use 
and  the  user  does  not  use  it,  it  has 
failed.  EMMA  uses  windows  to  relay  infor¬ 
mation  to  the  technician  and  accepts  infor¬ 
mation  via  menus.  Using  a  mouse,  the  tech¬ 
nician  is  able  to  enter  data  quickly  and 
accurately  without  having  to  learn  cryptic 
crjimsands.  The  majority  of  the  data  entered 
into  EMMA  by  the  technician  is  done  using 


the  mouse;  however,  some  Keyboard  input  is 
required.  Figure  3  shows  the  screen  of  a 
Symbolics  computer  running  EMMA. 

THE  GBU-1S  FIELD-LEVEL  EMMA  PROTOTYPE 


The  Rockwell  field-level  (phase  1) 
EMMA  prototype  was  designed  to  enhance  the 
field-level  maintenance  of  the  GBU-15  glide 
bomb  by  augmenting  the  field-level  test  set 
—  GJM-55.  All  references  to  the  word 
"EMMA"  in  this  section  refer  to  the 
Rockwell  GBU-15  version  of  EMMA  unless 
stated  otherwise.  EMMA  is  hosted  on  an  IBM 
PC/AT  compatible  computer  running  the 
expert  system  shell  M.l.  Although,  the  M.l 
language  is  primarily  rule-based,  EMMA  was 
developed  using  an  object  oriented 
approach.  The  rules  of  the  knowledge  base 
reference  objects  and  object  attributes. 
This  EMMA  did  not  support  the  capability 
for  an  automatic  mode  due  to  hardware 
limitations  thereby  leaving  only  the  manual 
mode  (i.e.,  no  connecting  cable).  Figure  4 
illustrates  the  major  components  of  the 
EMMA  system  and  how  they  are 
interconnected . 

Uncertainty  is  addressed  in  this 
version  of  EMMA.  When  EMMA  asks  the 
technician  for  information,  the  technician 
may  enter  "unknown*  as  a  response.  EMMA 
will  accomsodate  this  response  by  adapting 
its  reasoning  process  using  uncertainty. 
Uncertainty  is  handled  using  a  MYCIH-like 
representation  (i.e.,  using  a  combining 
function  that  produces  a  quantitative 
measure  of  uncertainty,  certainty  factors, 
that  is  bound  between  1  and  -1  with  1 
meaning  true  and  -1  meaning  false)  (Hayes- 
Roth  et  al.,  1983t93-96).  When  a  recommen¬ 
dation  is  displayed  to  the  technician,  the 
certainty  of  the  recommendation  is  also 
displayed  to  indicate  the  belief. 

The  GBU-15  EMMA  also  possesses 
explanation  capabilities.  The  technician 
may  ask  EMMA  for  an  explanation  or  help  at 
any  time.  EMMA  will  respond  with  either  an 
explanation  of  the  reasoning  process  or 
information  that  will  guide  the  technician 
through  the  consultation.  The  explanation 
capability  can  handle  queries  regarding  the 
reason  a  certain  conclusion  was  reached  or 
why  EMMA  is  asking  for  information.  As 
with  the  AIM-7F  EMMA,  the  GBU-15  EMMA  has 
two  levels  of  explanation  to  accommodate 
the  neeas  of  different  technicians.  The 
same  training  benefits  exist  in  the  GBU-15 
EMMA  as  the  AIM-7F  EMMA. 

EMMA  exploits  the  use  of  pull-down 
menus  and  function  keys  on  the  coaqniter  to 
make  it  as  user  friendly  as  possible.  The 
majority  of  technician  interaction  with 
EMMA  is  performed  using  the  keyboard.  The 
technician  typically  responds  to  EMMA 
questions  and  requests  with  short  answers 
thereby  reducing  the  probability  of  erro¬ 
neous  data  being  entered.  Figure  5  shows 
the  screen  of  the  coeqsuter  running  EMMA. 
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Figure  3.  A.IM-7F  EMMA  Screen 


BVALUATIOH  OF  THE  EMMA  PROTOTYPES 

Meaningful  evaluation  of  expert 
systems  has  been  an  often  discussed  but 
seldom  achieved  topic  within  recent  years. 
More  often  than  not  quantitative  metrice 
are  simply  not  available  or  meaningful  as 
an  evaluation  measure.  Since  an  expert 
system  encapsulates  the  knowledge  of  a 
given  expert  in  a  given  field,  the  effec¬ 
tive  evaluation  of  the  expert  system  may  be 
difficult  at  beat.  Validation  must  be  used 
to  justify  the  representation  1  avals  of 
expert  systems  (O'Keefe  et  al.,  1987). 

Validation  is  typically  considered  a 
part  of  evaluation,  and  evaluation  is  con¬ 
cerned  with  determining  the  comprehensive 
value  of  an  expert  system  (O'Keefe  et  al., 
1987).  Validation  should  not  be  confused 
with  verification.  "Validation  refers  to 
building  the  right  system  (that  is, 
substantiating  that  a  system  performs  with 
an  acceptable  level  of  accuracy),  whereas 
verification  refers  to  building  the  system 
'right'  (that  is,  substantiating  that  a 
system  correctly  implements  its  specifica¬ 
tions)"  (O'Keefe  et  al.,  1987).  Validation 
and  verification  will  be  addressed  in  this 
paper  as  they  apply  to  EMMA. 
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Verification  of  EMMA 

A  unique  aspect  of  the  verification  of 
the  EMMA  program  is  that  it  uses  DOD-STD- 
2167,  the  Defense  System  Software 
Development  standard,  to  develop  the  expert 
system  prototypes.  This  is  one  of  the 


Figure  4.  GBU-15  Field-level 
EMMA  Prototype 
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EMMA  CONFIG  NO:  33  TEST  NO:  0096  NOGO:  INV/M6  A1 


SRII  COMPONENT  CABLE  CERTAINTY 

INV/CONV  .  60% 

CONTROL  UNIT  (HARNESS)  UUT  INPUT  CABLE  053A20(BLU)  40% 

RECOMMENDATION: 

CHECK  THE  INPUT  CABLE  OF  THE  UNIT  UNDER  TEST.  IF  OK.  THEN 
RAR  INV/CONV 

JUSTIFICATION: 

A  RELATED  TEST  WITH  A  DIFFERENT  INPUT  CHANNEL  PASSED  AND 
THE  PARAMETERS  THAT  FAILED  WERE  APPROXIMATELY  ZERO:  THUS 
TIE  INPUT  CABLES  OF  THE  UNIT  UNDER  TEST  COULD  BE  THE 
CAUSE.  A  MORE  LIKELY  FAILURE  IS  THE  INV/CONV  BECAUSE  THE 
+  28  VDC  ELECTRONICS  IS  COMPLEX  AND  COULD  CAUSE  ZERO 
FAILURES  AS  INDICATED 


Figure  5.  GBU-15  EMMA  Screen 


firet  attempts  to  apply  this  standard  to 
the  development  of  an  expert  system.  The 
EMMA  program  has  shown  that  an  expert 
system  can  be  developed  using  DOD-STD-2167 
software  development  requirements.  With 
some  careful  tailoring  of  some  of  the 
documents,  this  standard  can  be  effectively 
used  to  document  the  program  and  provide 
the  program  manager  valuable  insight  into 
the  contractor's  software  development, 
testing,  and  evaluation  efforts.  The 
tailored  documents  were  slightly  altered  to 
accowdate  the  iterative  nature  of  expert 
system  development. 

Since  verification  must  determine 
whether  an  expert  system  correctly 
implements  its  specifications,  testing  must 
occur  in  order  to  validate  this 
requirement.  Again.  DOD-STD-2167  proved  to 
be  adequate  for  verification  testing  once 
extended.  Using  the  testing  documents 
called  out  in  this  standard,  the  correct 
implementations  of  the  specifications  for 
EMMA  were  verified.  Two  levels  of  testing 
occurred  to  accomplish  this  task.  First, 
the  knowledge  engineer  performed  informal 
testing.  This  testing  verified  the 
integrity  of  the  individual  computer 
software  units  before  the  units  were 
integrated  into  the  system  and  tested  as  a 
system.  Since  expert  system  development  is 
iterative  in  nature,  informal  testing 
essentially  occurs  throughout  development. 
The  knowledge  engineer  and  the  expert 
verify  the  expert  system's  behavior  thereby 
identifying  potential  corrections  and 
enhancements  irfiich  were  to  be  incorporated 
later  by  the  engineer.  Second,  formal 
testing  occurred.  An  independent  team 
performed  the  formal  testing  by  exsreising 
-MMA  using  test  plans  and  test  descriptions 
generated  using  DOD-STD-2167.  Both  forms 
of  tasting  identified  problems  which  were 
later  fixed.  This  generated  a  better 
e*P*ft  system  than  without  testing. 


Validation  of  EMMA 


Tne  validation  of  EMMA  is  addressed  in 
two  areast  performance  validation  (i.e., 
how  well  EMMA  performed),  and  the  human 
factors  aspect.  Both  areas  are  extremely 
important  to  the  success  of  an  expert 

_.Th?  followi“g  paragraphs  present 
the  validation,  methodology,  and  results  of 
the  two  EMMA  prototypes. 


As  with  most  expert  systems,  the  ulti¬ 
mate  measure  of  success  is  determined  when 
the  system  is  used  by  the  end  users  (in 
.th®  field-level  munition  techni- 
cian).  This  is  the  approach  taken  with  the 
smma  program.  Both  contractors  took  their 
respective  prototypes  to  Air  Force  bases  in 
Milch  their  selected  munition  is  used  and  a 
rield-level  maintenance  capability  exists. 
This  allowed  the  prototypes  to  be  evaluated 
in  an  actual  field  test  environment. 


considering  several  alterna¬ 
tives,  both  contractors  decided  to  use  a 
toggle  switch  box  to  insert  faults  into  a 
Known  good  missile.  This  approach  was 
necessary  since  it  was  feared  that  the 
maintenance  squadrons  in  the  field  might 
not  have  a  sufficient  number  of  faulty 
munitions  during  the  evaluation  period. 
These  faults  were  induced  by  the  user  by 
simply  toggling  one  of  the  switches  which 
in  turn  would  disturb  one  or  more  signals 
'Ji^hin  th®  munition.  The  faults  were 
d . ined  by  the  domain  expert  in  conjunction 
with  the  maintenance  expert  (i.e.,  the 
®xP*rts  select  various  representative  tests 
from  the  test  set)  such  that  they  would 
adequately  exercise  the  various 
characteristics  of  the  EMMA  prototypes 
which  included  the  resequencing  logic,  the 
explanation  capability,  and  the  fault 
isolation  logic. 
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The  AIM-7F  EMMA  Evaluation.  Raytheon 
took  their  field  level  AIM-7F  prototype  to 
the  325th  Equipment  Maintenance  Squadron 
(EMS)  at  Tyndall  AFB,  Florida  for  an 
evaluation  period  that  began  on  8  June  1987 
and  concluded  on  12  June  1987. 

The  argument  could  be  made  that  EMMA 
should  accurately  diagnose  all  the  induced 
faults  since  the  expert  system  and  the 
faults  were  derived  from  the  same  source  — 
the  domain  expert.  In  order  to  demonstrate 
the  robustness  of  EMMA,  ar.  additional 
evaluation  methodology  was  used.  Two 
faulted  missiles  were  saved  by  the  EMS 
prior  to  the  evaluation.  These  missiles 
had  previously  failed  testing  using  the 
AN/DSM-162  test  set.  However,  the  fault 
d^ta  for  these  missiles  were  not  released 
by  the  EMS  personnel  until  after  the  EMMA 
evaluation.  A  third  missile  became  avail¬ 
able  during  the  EMMA  evaluation  by  failing 
a  flight  line  test  during  prelaunch  tuning. 
This  missile  was  an  excellent  exercise  for 
the  EMMA  prototype  since  it  was  not  pre¬ 
viously  tested  by  the  AN/DSM-162  test  set. 
Its  fault  was  unknown  to  everyone  present 
at  the  evaluation.  All  three  missiles 
(referred  to  as  "mystery  missiles"  due  to 
their  unknown  past)  contained  faults  un¬ 
known  to  EMMA,  the  domain  expert,  and  the 
maintenance  technicians  as  well. 

Four  amnition  maintenance  technicians 
from  the  EMS  at  Tyndall  were  used  for  the 
evaluation.  Two  technicians  were 
classified  as  novice  with  little  experience 
with  the  AH/DSM-162  test  set  and  its 
associated  operating  procedures.  The  other 
two  technicians  were  classified  as  experts 
with  a  substantial  background  in  using  the 
AH/DSM-162  test  set.  Two  teams  of  two 
technicians  were  created  consisting  of  one 
expert  and  one  novice.  One  team  (hereafter 
referred  to  as  the  EMMA  team)  received 
extensive  training  on  the  operation  of 
EMMA.  The  other  team  (hereafter  referred 
to  as  the  non-EMMA  team)  was  not  trained  on 
the  EMMA  system  and  served  as  a  baseline 
for  the  evaluation. 

Twelve  faults  were  inserted  into  a 
known  good  missile  using  the  toggle  switch 
box.  The  faults  were  diagnosed  by  the  EMMA 
team  using  the  EMMA  system  and  the  non-EMMA 
team  using  just  the  AH/DSM-162  test  set  and 
the  applicable  TO.  Performance  of  the  two 
teams  was  based  on  the  level  of  expertise 
of  the  operator,  duration  of  test,  and  the 
ability  to  diagnose  the  fault  accurately. 

The  results  of  this  evaluation 
exercise  were  very  promising.  There  were 
three  significant  results  derived  from  the 
evaluation.  First,  the  EMMA  system 
operated  by  the  EMMA  team  was  able  to 
consistently  diagnose  the  fault  quicker 
than  the  non-EMMA  team  using  just  the 
AH/DSM-162  and  the  technical  orders 
regardless  of  the  experience  level  of  the 
EMMA  operator.  A  time  ravings  of  20%  was 
seen  with  the  novice  usii.g  EMMA  versus  the 
expert  using  the  AN/DSM-162.  Second, 
novice  technicians  using  the  EMMA  system 


significantly  outperformed  (better  fault 
diagnoses)  novice  technicians  using  just 
the  AN/DSM-162  and  performed  33%  better 
than  expert  technicians  using  just  the 
AH/DSM-162.  Finally,  EMMA's  explanation 
capabilities  significantly  enhanced  the 
abilities  of  the  EMMA  team  to  determine  the 
reason  behind  each  fault. 

Once  EMMA'S  abilities  were  exercised 
using  the  induced  faults,  EMMA  was  pitted 
against  the  mystery  missiles  again  with 
excellent  results.  The  EMMA  team  using 
EMMA  correctly  isolated  the  faults  in  all 
three  missiles.  Only  after  EMMA  diagnosed 
the  faults  was  the  previous  testing  data  on 
the  two  saved  missiles  released.  EMMA's 
diagnosis  was  consistent  with  this  data. 

User  acceptance  of  EMMA  was  outstand¬ 
ing.  In  fact,  the  technicians  accepted 
EMMA's  diagnosis  of  the  missile  from  the 
flight  line  and  said  they  would  have,  if 
allowed,  sent  the  missile  to  the  depot  with 
no  further  testing  using  the  AN/DSM-162 
test  set.  This  exemplifies  EMMA's 
acceptance  by  the  EMS  maintenance  personnel 
at  Tyndall  AFB.  The  technicians  found  the 
system  to  be  very  user-friendly.  The  mouse 
and  the  use  of  menus  made  the  system  easier 
to  use  than  the  bulky  and  cumbersome  TOs. 
Also,  the  explanation  capability  proved  to 
be  an  effective  training  mechanism. 

The  GBU-15  EMMA  Evaluation.  Rockwell 
evaluated  their  GBU-XS  EMMA  at  the  4th 
Equipment  Maintenance  Squadron  located  at 
Seymour  Johnson  AFB,  Horth  Carolina  during 
the  period  of  22  June  through  29  June  1987. 

Four  maintenance  technicians  were  used 
in  the  evaluation  of  the  EMMA  prototype. 
Two  technicians  were  considered  experts 
with  several  years  of  experience  with  the 
GBD-15  test  environment.  The  remaining  two 
technicians  were  considered  novices  with 
less  than  6  months  of  experience.  Another 
important  distinction  between  the  expert 
and  novice  technicians  is  the  fact  that  the 
expert  technicians  owned  personal  computers 
and  therefore  were  familiar  with  how 
computers  operate,  whereas  the  novice 
technicians  did  not  own  computers  and  had 
never  used  a  computer  before  the  EMMA 
evaluation.  All  four  technicians  were 
trained  cn  how  to  use  the  EMMA  system. 
After  this  brief  training,  the  technicians 
felt  very  comfortable  using  the  system. 

Twenty-two  simulated  faults  were 
induced  into  the  known  good  munition  with 
the  intent  of  evaluating  EMMA's  capabili¬ 
ties  to  handle  the  following  five  areas: 
resolution  of  ambiguities  between  major 
shop  replaceable  units  (SRU),  referencing 
lower  configuration  testing  to  facilitate 
further  component  resolution,  distin¬ 
guishing  between  a  cable  failure  and  a 
circuit  card  assembly  (CCA)  gain  failure, 
resolution  of  ambiguities  between  CCA's, 
and  recognizing  operator  errors  or  test  set 
problems.  Six  of  the  twenty-two  induced 
faults  were  in  the  all-up-round  (AUR) 
configuration  (i.e.,  the  test  was  performed 


while  the  GBU-15  munition  was  completely 
intact).  The  remaining  sixteen  faults  were 
in  the  control  module  stand  alone 
configuration.  EMMA  was  able  to  handle 
these  five  areas  by  analysing  additional 
test  parameters  as  well  as  instituting  and 
analyzing  tests  related  to  the  failed  test. 

The  diagnostic  results  of  the  induced 
faults  showed  substantial  time  savings  in 
fault  isolation  and  increased  diagnostic 
capabilities.  While  the  munition  was  in 
the  control  module  stand  alone  configura¬ 
tion,  a  time  savings  of  40%  was  seen  over 
conventional  testing  with  the  GJM-55.  When 
the  munition  was  in  the  AUR  configuration, 
EMMA  was  able  to  provide  up  to  74%  time 
savings.  This  is  due  to  EMMA' a  capability 
to  resolve  failures  while  the  munition  is 
in  the  AUR  configuration  thereby  saving  the 
technician  from  having  to  performing 
testing  in  stand  alone  configuration. 

The  G  JM  —  5  5  test  set,  in  some 
situations,  will  recommend  more  than  one 
suspected  failure.  This  group  of  failures 
is  called  an  ambiguity  group  since  the  teat 
set  cannot  resolve  any  further  than  this 
group.  EMMA  was  able  to  break  up  ambiguity 
groups.  EMMA  also  considered  the 
possibility  of  a  cable  harness  failure  or 
the  test  set  is  failing.  Based  on  these 
capabilities  EMMA  was  able  to  significantly 
improve  fault  isolation  aa  seen  by 
diagnosing  the  22  simulated  faults. 
EMMA  added  a  wiring  harness  check  to  S0%  of 
all  tests.  EMMA  deleted  a  CCA  from  an 
ambiguity  group  40%  of  the  time  thereby 
reducing  the  number  of* CCA  to  be  considered 
during  testing.  EMMA  added  a  CCA  to  an 
ambiguity  group  30%  of  the  time  to  insure 
all  potential  CCA* a  are  considered  during 
the  testing.  This  suggests  that  the  test 
set  did  not  always  consider  all  potential 
CCA's.  Finally.  EMMA  exchanged  one  suspect 
CCA  in  an  ambiguity  group  for  another  CCA 
10%  of  the  time.  The  ability  to  manipulate 
the  ambiguity  group  to  benefit  fault 
isolation  was  demonstrated  by  EMMA  and 
proved  to  be  an  effective  fault  isolation 
technique.  These  results  directly  support 
the  time  savings  previously  mentioned. 

The  GBU-15  EMMA  prototype  also 
received  accolades  for  its  user 
friendliness.  The  techniciana  used  EMMA 
with  comfort  and  found  several  items  to  be 
particularly  laudable.  Among  these  items 
was  the  understandability  of  EMMA.  The 
explanation  capability  provided  easy  to 
understand  responses.  Another  aspect  they 
found  beneficial  was  the  addition  of  the 
internal  wiring  harness  check  as  one  of  the 
reasons  for  a  fault  since  this  check  is 
relatively  "inexpensive”  to  perform  and  can 
prevent  unnecessary  and  potentially  costly 
future  testing.  The  training  potential  of 
EMMA  was  also  mentioned  as  one  of  its  major 
assets  with  the  shortage  of  skilled  techni- 
:ians  in  the  munition  maintenance  field. 


EMMA  PHASE  2 

Both  contractors  are  currently  in 
phase  2  of  the  EMMA  program.  As  previously 
mentioned.  phase  2  focuses  on  the 
maintenance  of  tactical  munitions  at  the 
depot  level.  More  specifically,  Raytheon 
is  focusing  on  the  depot-level  maintenance 
of  the  AIM-7F.  Rockwell  is  using  the  GBU- 
15  as  its  depot- level  maintenance  munition. 
Phase  2  is  a  natural  extension  of  phase  1 
since  field-level  faults  are  sent  to  the 
depot  for  repair.  The  prototypes  developed 
during  the  phase  2  effort  will  be  more 
detailed  extensions  of  the  phase  1 
prototypes  with  one  exception;  the  phase  2 
prototypes  will  augment  the  depot-level 
test  sets.  The  depot  test  set  for  the  AIM- 
7F  is  the  AN/DPM-22  test  set.  The  GBU-15 
depot-level  test  set  is  CATS  (Calculator 
Automatic  Test  Station). 

Tho  depot- level  prototypes  will  be 
implemented  on  the  same  computer  hardware 
using  the  same  expert  system  shells  as  the 
field-level  prototypes.  However,  one 
difference  between  the  field  and  depot 
prototypes  for  both  contractors  is  the 
interface  between  the  test  set  and  the  EMMA 
computer.  The  Raytheon  interface  will  only 
support  one-way  cossnunication  from  the  test 
set  to  the  EMMA  computer  due  to  test  set 
limitations.  This  is  different  than  the 
two-way  communication  of  the  field 
prototype.  Rockwell  is  using  a  two-way 
communication  interface  between  the  test 
set  and  the  EMMA  computer  whereas  the  field 
prototype  interface  was  manual.  Both 
prototypes  will  again  incorporate  an 
explanation  capability  for  the  technicians. 

The  evaluation  of  the  depot  prototypes 
will  follow  the  same  methodology  used  in 
phase  1.  Each  prototype  will  be  evaluated 
at  the  actual  depot  location  by  actual 
depot  technicians.  Once  again,  both 
evaluations  are  scheduled  to  last  5  days 
and  are  scheduled  to  occur  in  February 
1989.  The  AIM-7F  prototype  will  be 
evaluated  at  the  Naval  Aviation  Depot 
(NAVAVNDEP)  in  Alameda,  California.  The 
GBU-15  prototype  will  be  evaluated  at 
Rockwell's  Missile  Systems  Division  in 
Atlanta,  Georgia  since  an  organic  depot 
capability  currently  does  not  exist. 

FOLLOW-ON  PROGRAMS  TO  EMMA  ~  EMMA  2 

A  follow-on  program  will  be  initiated 
in  early  1990  —  EMMA  2.  The  primary 
thrust  of  EMMA  2  is  to  develop  an  expert 
system  that  is  capable  of  diagnosing  a 
family  of  tactical  munitions  at  the  depot 
level.  The  current  EMMA  is  limited  to  one 
munition  per  prototype.  EMMA  2  will 
attempt  to  expand  the  current  prototype 
capabilities  to  include  multiple  munitions 
from  the  same  family  (e.g.,  AIM  family,  GBU 
family,  surface-to-air  family,  etc.).  EMMA 
2  will  draw  on  the  best  features  of  all 
prototypes  developed  in  the  two  phases  of 
EMMA  to  derive  a  robust  system.  Some  of 
the  areas  of  interest  that  are  being 
considered  for  EMMA  2  include:  de«p 
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reasoning  techniques,  speech,  knowledge 
acquisition  tools/aids,  and  generic 
knowledge  base  creation  and  maintenance. 

OTHER  ISSUES /OBSERVATIONS 
Current  Maintenance  Philosophy 

The  current  munition  maintenance  phi¬ 
losophy  of  the  Tactical  Air  Command  (TAC) 
for  field  maintenance  is  that  of  fault 
detection  (go/nogo  testing).  Depending  on 
the  munition,  if  a  fault  does  occur  in  the 
field,  the  suspected  faulty  section  of  the 
missile  is  sent  to  the  depot  for  repair. 
One  of  the  driving  factors  of  this  philoso¬ 
phy  is  the  shortage  of  skilled  maintenance 
technicians  in  field-level  maintenance. 

Since  training  these  technicians  is 
costly,  TAC  decided  to  eliminate  an  Air 
Force  Specialty  Code  (AFSC)  for  munition 
maintenance.  The  deleted  AFSC,  316X1L,  was 
an  electronics  munition  maintenance 
specialist.  With  this  specialist  no  longer 
available,  munition,  not  electronic 
munition,  specialist  are  diagnosing  today's 
munitions.  This  tends  to  create  problems 
since  the  munition  specialists  are 
typically  not  adequately  trained  to 
-diagnose  the  electronically-sophisticated 
munitions  of  today. 

EMMA  is  capable  of  providing  the  nec¬ 
essary  training  and  assistance  to  munition 
maintenance  technicians.  Using  the  expla¬ 
nation  capabilities  of  EMMA,  a  technician 
can  quickly  become  skilled  at  diagnosing 
the  munition.  Since  EMMA's  knowledge  is 
gleaned  from  diagnostic  and  design  experts, 
the  novice  munition  technician  using  EMMA 
will  effectively  be  performing  as  if  he  has 
an  expert  maintenance  technician,  the 
designer  of  the  test  set,  the  munition 
designer,  and  an  instructor  looking  over 
his  shoulder  during  the  diagnosis.  Another 
aspect  of  EMMA  that  should  reduce  overall 
maintenance  costs  is  its  ability  to 
diagnose  to  a  greater  functional  level  than 
existing  test  sets  used  by  today's 
technicians.  This  should  significantly 
reduce  the  costs  associated  with  shipping 
faulty  munitions  to  the  depot,  since  more 
faults  can  be  isolated  at  the  field. 

The  munition  depots  are  currently 
responsible  for  the  majority  of  munition 
repairs.  The  field  has  very  limited  repair 
capabilities.  This  results  in  increased 
coats  for  overall  maintenance  of  a  munition 
system.  The  obvious  cost  is  transportation 
of  the  munition  between  the  field  and  the 
depot.  However,  the  less  tangible  and 
potentially  more  significant  cost  is  that 
of  having  the  munition  out  of  the 
inventory.  This  effectively  reduces  the 
number  of  missiles  available  for  exercises 
or  conflict. 


Future  Maintenance  Systems 

EMMA  prototypes  have  proved  the 
feasibility  of  applying  AI  to  munition 
maintenance.  In  future  weapon  systems, 
maintenance  expert  systems  should  evolve 
with  the  weapon  system  instead  of  after  the 
fact.  This  would  allow  the  expert  system 
to  capture  knowledge  about  the  weapon  as  it 
is  developed.  Furthermore,  the  expert 
system  should  be  incorporated  directly  into 
the  ATE  instead  of  augmenting  the  ATE  with 
a  separate  computer  system.  This  should 
make  weapon  systems  of  the  future  more 
supportable  and  maintainable  by  considering 
the  maintenance  aspect  early  in  the  weapon 
life  cycle. 

SUMMARY 

Tactical  munition  maintenance  of  today 
has  problems.  EMMA  is  an  attempt  to  re¬ 
lieve  some  of  these  problems  by  applying 
artificial  intel ligence/ expert  system  tech¬ 
nology.  The  results  of  EMMA  indicate  that 
this  approach  to  munition  maintenance  has 
significant  potential  for  future  tactical 
maintenance  systems. 

Corporate  knowledge  retention  is  one 
of  the  premium  benefits  of  EMMA.  Since 
EMMA  is  updated  easily  and  it  never 
"forgets*  knowledge,  EMMA  is  an  excellent 
tool  for  storing  corporate  knowledge  as 
technicians  come  and  go  thereby  making  EMMA 
a  better  diagnostic  entity  than  any  one 
expert.  Also,  EMMA  provides  consistent, 
high  quality  diagnosis  since  it  never  has  a 
"bad"  day  as  contrasted  with  technicians. 
Rapid  fault  isolation  and  efficient 
manpower  utilization  are  two  more  benefits 
of  using  EMMA.  These  benefits  provided  by 
EMMA  will  result  in  substantial  mission 
payoffs.  Weapon  system  downtime  wi.ll  be 
decreased  as  well  as  personnel  requirements 
and  training  time.  However,  the  most 
significant  payoff  is  the  increase  in  the 
reliability  of  munition  maintenance 
procedures . 

The  technology  has  proven  itself.  The 
Government  and  contractors  must  now 
concentrate  their  efforts  on  getting  it 
integrated  into  the  next  generation  ATE. 
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